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Abstract 


In item-response theory, if a latent-structure model has an ability variable, then elementary 
information theory may be employed to provide a criterion for evaluation of the information the 
test provides concerning ability. This criterion may be considered even in cases in which the 
latent-structure model is not valid, although interpretation of the criterion is more complex in 
this case. It is also possible to consider reliability results for inferences about the ability variable. 
Use of sampling to develop required parameter estimates is straightforward. Applications of the 
criterion in zero-, one-, and two-parameter logistic (OPL, 1PL, and 2PL) examples are provided. 
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Given an item-response model, information theory can be employed to assess the information 
concerning an ability parameter that is provided by the data. The argument is the same as the 
argument used to define the information provided by an experiment (Lindley, 1956), although 
some care must be taken to treat the issue of false models. In this paper, a random ability 
parameter is generated that corresponds to a conventional ability parameter if a model is true 
but is shown to be definable and estimable even if the model is not valid. In Section 1, the basic 
results from information theory are provided, and suitable estimation procedures are developed. In 
addition, the zero-, one-, and two-parameter logistic (OPL, 1PL, and 2PL) models to be examined 
are described for the case of a model with a normal ability distribution and for a model with a 
multinomial ability distribution. Section 2 summarizes results obtained by analysis of a 45-item 

TM 

right-scored multiple-choice writing examination from the Praxis series that has previously been 
employed (Haberman, 2005) to illustrate use of latent-structure models. This example is quite 
suitable for analysis in that the data are easily shown not to satisfy any standard item response 
model, but standard 1PL models appear to provide reasonable approximations of the observed 
distribution of item responses for the 8,686 examinees. Section 3 provides some conclusions. The 
results on the whole suggest that efforts to estimate ability parameters are surprisingly successful 
even when models are not valid. 

1 Information Concerning Ability 

In this section, information theory is used to examine the information provided concerning 
ability variables associated with item-response models. Consider binary random item responses 
X v i by examinee v to item i, 1 < i < /, 1 < v < n, where the number / of items is at least 2 
and the number n of examinees is at least 2. For each examinee v, 1 < v < n, and each item i, 

1 < i < /, let X vl have possible integer values 0 or 1, where X v { is 1 if a correct response is given 
to by examinee v to item i and X v i is 0 otherwise. The responses for examinee v may be described 
by X„, the /-dimensional response vector with coordinates X V i, 1 < * < /. If A denotes the set 
of /-dimensional vectors x with all coordinates 0 or 1, then X is the set of possible values of X„. 
It is assumed in this paper that the probability p(x) that X„ = x is positive for each possible x 
in X, and it is assumed that the response vectors X,,, 1 < v < n, are mutually independent and 
identically distributed. To simplify notation, the convention is adopted that Xi is denoted by X, 
and X\i is denoted by X t . 
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1.1 Latent-Structure Models 

This paper is concerned with the use and interpretation of latent ability vectors for item 
responses when the models are not necessarily true. In the models under study, a single 
latent variable is used; however, generalization to multiple latent variables is straightforward. 
Illustrations are based on the 2PL model (Hambleton, Swaminathan, & Rogers, 1991), but the 
methods proposed apply much more widely. The methodology proposed can be employed both 
with ability variables with a normal distribution or with ability variables with a multinomial 
distribution. 

For any model under study, it is assumed that some real latent variable 9 V is associated 
with each examinee v. It is assumed in each model under consideration that the pairs (X,,, 9 V ), 

1 < v < n, are mutually independent and identically distributed, and the local independence 
assumption is made that the responses X V i, 1 < * < I, of subject v are conditionally independent 
given 9 V . The convention is adopted that 9\ is denoted by 9. Associated with each item i is an 
item characteristic curve (ICC) P % such that Pi(uj) is the probability that X t = 1 given that 9 = uj. 
In all models in this paper, the 2PL assumption is made that 

Pi{io) = [1 + exp(—ajo; + y*)] -1 (1) 

for an unknown positive real a t , the item discrimination of item i, and for a real unknown 77 The 
ratio ^ = ji/ai is the item difficulty for item i. In the 1PL (Rasch) models under study, each a* is 
assumed equal. In the OPL (binomial) models under study, each a* is assumed equal and each 7 \ is 
assumed equal. In the normal latent-variable models under study, 9 is assumed to have a standard 
normal distribution. In the latent-class models under study, an integer K > 2 and distinct real 
numbers 77, 1 < k < K, are specified, 9 V can only assume the values 77, 1 < k < K, and 9 V = Tk 
with an unknown but positive probability. 

In any of these latent-variable models, if the model is correct and the model parameters are 
known, then Bayes’ theorem may be employed for inferences about the latent variable 9 V based 
on the response vector X„ (Bock & Aitkin, 1981). In practice, maximum-likelihood estimates of 
model parameters may be used instead of the model parameters themselves if the sample size is 
large. 

If the model is not correct, then the meaning of inferences concerning the 9 V is not obvious. 
Indeed, it is not even obvious what the 9 V are. In this paper, a procedure based on information 
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theory is employed to construct random variables 9 V with conditional distributions given the 
responses X„ that are of the same form as the corresponding conditional distributions obtained 
if the model holds. These random variables 9 V may be constructed by use of a random number 
generator capable of providing independently distributed uniform random numbers U v for 
1 < v < n, where the U v are independent of the observed responses Xj A , and have range (0,1). 
Information theory is used to evaluate the strength of the relationship of these 9 V to the observed 
examinee responses X„ of examinee v and to evaluate the reliability of a standard estimate of 9 V . 
Maximum likelihood is readily used to estimate all needed parameters and measures. 

1.2 Models and Information Theory 

To develop the variables 9 V , information theory is employed to obtain approximations to the 
response probabilities p(x), x in X that are consistent with the latent-variable model under study 
(Gilula & Haberman, 1994, 1995; Savage, 1971; Shannon, 1948). To apply information theory, 
some preliminary definitions are needed. Let p denote the function on X with value p(x) for x 
in X, so that p determines the distribution of each response vector X,, . Let S be the simplex of 
nonnegative real functions r on X with sum r(x) = 1, so that p is in S and to any r in S 

corresponds a random vector Y with values in X such that Y = x with probability r(x) for each 
x in X. For the model under study, let T be the set such that p is in T if, and only if, the model 
holds. For example, in the case of a normal 2PL model, r is in the set T if, and only if, for each x 
in X, 

r(x) = J r{x\uj)^>{u)duj , (2) 

where 

i 

r(x \u) = (3) 

i= 1 

4> is the density function of the standard normal distribution, and ( 1 ) holds for some real a* > 0 
and real 7 i, 1 < i < /. Observe that if the normal 2PL model holds for pairs (X t) , 9 V ) and r = p, 
then it is clear that 9 V has a standard normal distribution and (1), (2), and (3) hold for some real 
Oi > 0 and 7 i, so that p is in T. 

On the other hand, if p is in T, then (1), (2), and (3) hold for r = p for some real a* > 0 
and real 7 *. Given the uniform random numbers U v , 1 < v < n, random variables 9 V may be 
constructed so that the pairs (X„, 9 V ) are mutually independent and the conditional density of 9 V 
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given X„ = x is 


(4) 


/(w|x) = [p(x)] V(w)r(x|w) 

for to in R. It suffices to define the strictly increasing and continuous distribution function 

/ U) 

f{y\x)dv (5) 

-OO 

for u) real and to let 

F(6 V |X„) = U v (6) 

(Rao, 1973, p. 87). The marginal density of 9 V is 

Y /H X M X ) = <K W )> 

xeA” 

so that 9 V has a standard normal distribution. Bayes’ theorem implies that the conditional 
probability that X„ = x given 9 V = lo is r(x|u;). Thus the resulting pairs (X V ,9 V ) satisfy the 
normal 2PL model. 

Similar arguments apply to other models under study. For example, in the 2PL latent-class 
case, r is in T if, and only if, (1) holds for some a* > 0 and real 7j, 

K 

r(x) = y^7r fc r(x|Tfc) 
k =1 

for 77 > 0, 1 < k < K , such that J2t=i Tfc = 1, and (3) holds for 00 real. If the latent-class 2PL 
model holds for pairs (X,,, 9 V ) and r = p, then 9 V = 77. with probability 77 > 0, 1 < k < K , where 
Sfc=i 77 k = 1) an< 4 (1), (2), and (3) hold for some real a* > 0 and 7j, so that p is in T. 

On the other hand, let p be in the set T defined for the latent-class 2PL model. Then (1), 
(2), and (3) hold for r = p for some real a* > 0 and real 7j and for some 77, > 0, 1 < k < K, such 
that ^2k=i 77 k = 1- Construct a random variable 9 V from the uniform random number U v such 
that the conditional probability that 9 V = 7 given X„ = x is equal to 

fin c|x) = [p(x)] _1 7r fc r(x|r fc ) (7) 

for 1 < k < K. If the 7 are increasing in k, then this construction is accomplished by letting 
9 V = k by use of the distribution function values 

k 

F(r fc |x) = ^/(7|x) (8) 

h= 1 
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for 1 < k < K. Let 9 V = t*, for 1 < k < K if U v < Ffa |X„) and either k = 1 or U v > F(jk- i|X„). 
The probability that 9 V = t\. is 717. for 1 < k < K, and the conditional probability that X,, = x 
given 9 V = r fc is r(x|r fe ). 

To examine the more general case in which the model under study need not be true, consider 
probability prediction of X„ by use of a logarithmic penalty function (Gilula & Haberman, 1994, 
1995, 2000). If the prediction r in S is employed and if the penalty is — logr(x) for X,, = x in A, 
then the expected penalty per item is 

J( r) = -/ _1 £(logr(X)) = -I -1 ^p(x)logr(x), 

xeA” 

where OlogO = 0 (Haberman, 2005). The minimum of J( r)) is achieved if, and only if, r = p, and 

ff(x) = J( p) = -r l J^p(x)iogp(x) 

xe* 

is the entropy per item of X. To measure the discrepancy between p and the probability model 
that p is in T, consider the minimum H*(X.) of J(r) for r in T. This minimum cannot be less than 
H(X), and 77* (X) = 77(X) if the model holds. Thus a measure of model error is the minimum 
discriminant information D(X) = H*(X) — H (X) for distinguishing between the true probability 
vector p and a probability vector r in T (Haberman, 1982; Kullback, 1968). The measure D(X) is 
nonnegative. If H(X) is positive, then the model is not valid. One may regard D(X) as a measure 
of the average increase per item in log penalty due to use of the model that p is in T rather than 
some model that is correct. 

If p* in T satisfies J(p*) = LP(X), then p* is an optimal approximation to p within T 
according to the criterion of expected log penalty. If the model holds, then p* = p. In general, 
p* satisfies conditions quite similar to maximum-likelihood equations for log-linear models in the 
case of indirect observation (Haberman, 1979, chap. 10) or maximum-likelihood equations for 
exponential families in which observation is incomplete (Dempster, Laird, &; Rubin, 1977). These 
conditions may be described in terms of moments that involve a vector X* with the same possible 
values as the observed vector X and a random variable 9*. As required in a latent-structure model, 
the coordinates X* of X* are conditionally independent given 9* , and X* = x with probability 
P*(x). Additional moment restrictions and distribution restrictions are imposed on 9* and X* 
by the specific model under consideration. In the case of the normal 2PL model, for each item 
i, positive a* and real 7$ exist such that the conditional probability that X* = 1 given 9* = u 
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is Pi(u), where Pi(w) satisfies (1). In addition, 9* has a standard normal distribution. For each 
item i, the moment conditions are added that X* and X, t have the same expected values, and the 
expected value E(X*9*) is equal to the expected value of E(Xid), where 9 is a random variable 
such that the conditional distribution of 9 given X = x is the same as the conditional distribution 
of 9* given X* = x for each x in X. 

In the latent-class 2PL model, it remains the case that, for each item i, positive a* and real 
7 i exist such that the conditional probability that X* = 1 given 9* = uj is Pi(uj), where Pi(co) 
satisfies (1); however, 9* now assumes only the values 77, 1 < k < K, and, for 1 < k < K, the 
probability is positive that 9* = 77. It also remains the case that, for each item i, X* and Xj have 
the same expected value, and E(X*9*) = E(Xi9), where the conditional distribution of 9 given 
X = x is the same as the conditional distribution of 9* given X* = x for each x in X. The added 
condition is imposed that 9* and 9 have the same distribution. 

In all cases, the ability variables 9 V under study will be functions of the uniform random 
variable U v and the observed responses X„, and the conditional distribution of 9 V given X. u = x 
will be the same as the conditional distribution of 9* given X* = x. If the proposed model actually 
holds, then (9 V , X,,) will have the same joint distribution as (0, X). In the normal 2PL example, 
the 9 V are still obtained by use of (1), (2), (3), (4), (5), and (6). In the latent-class 2PL example, 
the 9 V are still obtained by use of (1), (2), (3), (7), and (8). 

1.3 Evaluation of the Ability Variable 

The variable 9 V constructed for each examinee v need not satisfy the local independence 
requirement of the latent-structure model if the latent-structure model itself does not hold. 
Nonetheless, 9 V may still be an effective predictor of the response vector X,,. The effectiveness of 
9 V may be evaluated by use of information theory in conjunction with the latent-structure model. 
Given Q v . the appropriate probability prediction of X„ is the conditional probability function 
p*(0„) on X with value p*(x|0„) at x in X. The expected log penalty per item is 

H*(X\9) = —I~ L E(\ogp* (X|0)). 

If the model holds, then H*(X.\9) is the conditional entropy per item H(X.\9) of X given 9. Given 
conditional independence of the X* given 9 *, 

1 

H*(X\9)=r 1 Y,H*(X l \0), 

i= 1 
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where 


H*(Xi\9) = -E(Xil og Pi(0) + (1 - Xi) log[l - Pi(d)]). 

If the model holds, then H*(X{\9) is the conditional entropy H(Xi\9) of response Xi given the 
variable 9. 

As shown in this section, 

H*{X\9) < H*(X). (9) 

The difference 

A(X\0) = H*(X)-H*(X\6) (10) 

provides a measure of the information per item on X provided by 9. 

1.4 The Latent-Class Case 

To verify (9), it is helpful to distinguish between the latent-class models and the normal 
models. In the latent-class models, the logarithmic penalty arguments for X can also be applied 
to (X,9) and 9. Let fWk |x) denote the conditional probability that 9* = Tfc given that X* = x, so 
that 

P* ( x , Tfc) =p*(x)/*(r fc |x) 

is the joint probability that (X*, 9*) = (x, t\.) for x in X and 1 < k < K . The expected log penalty 
per item for the corresponding probability prediction for (X, 9) is then 

H*(X,9) = -r 1 E(hgp*(X,9)). (11) 

If the model holds, then H*(X,9) is just the joint entropy H(X,9) of X and 9, so that 
H*(X, 9) > H(X, 9), with equality if the model holds. 

The probability prediction for 9 = t*, given that X = x is f*{jk |x), so that the expected log 
penalty for prediction of 9 by X is 


H*(6\X) = -£7(log/*(0|X)). (12) 

In this instance, the definition of 9 implies that H*(9 |X) is also the conditional entropy H{9 |X) of 
9 given X. It follows that 

H*(X,9) = H*(X) + I~ 1 H{9\X). (13) 
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An alternative decomposition to (13) may be created by use of the marginal distribution of 9 
predicted by the model. Let 9* = Tk with probability 7rj£. Then 

P*(x,r fe ) = 7TfeP*(x|r fc ). 

Consider the probability prediction of 9 = Tk by 7r£. Observe that the actual probability that 
9 = Tk is 7 Tk = E(f*(rk |X)). The expected log penalty is then 

K 

H*(9) = -E{logir* e ) = - J^7r fc log7r^, (14) 

fc=i 

and the entropy of 9 is 

K 

H{9) = - y^7 Tk log 7 Tk. 

k =1 

Thus H*{9) > H(9), with equality if the model holds. 

A new decomposition of the joint expected log penalty iL*(X, 9) is now available, for 


LP (X, 9) = H*(X\9) + I~ 1 H*{9). 


(15) 


By (13) and (15), 


A(X|0) = I~ 1 A(9 |X), 


where 

A(0|X) = H*(9) -H(9\X). 


(16) 

(17) 


Because H*(9) > H(9) and H(9) > H{9 |X) (Shannon, 1948), it follows that A(X|0) and A(0|X) 
are nonnegative. The constraint that a* > 0 and the constraint that each Tk > 0 imply that 9 and 
X are not independent, so that H{9) > H(9 |X). Thus A(X|0) and A(0|X) are positive. 

The measures A(X|0) and A(0|X) indicate the strength of the relationship between 9 and X. 
The perspectives are a bit different. In the case of A(X|0), one considers the improvement per 
item in probability prediction of X achieved by use of 9. In the case of A(0|X), one considers the 
improvement in probability prediction of 9 by use of the response vector X. 


1.5 The Normal Case 


Verification of (9) in the normal case is a bit different due to the continuity of 9. Some 
penalties based on log probabilities are replaced by log densities (Gilula & Haberman, 2000), and 



penalties are not necessarily nonnegative. The conditional density f*(u> |x) of 9* given X* = x is 
considered. The product 

P*(x,w) = p*(x)f*(u |x) 

is used for prediction of (X, 6) for x in X and to real, and the log penalty per item for the 
prediction is 

H*(-X,9) = -r 1 E(hgp*(X 1 9)). (18) 

The density prediction for 9 given X = x is the function with value f*{9 |x) at 9 real, and 
the expected log penalty for prediction of 9 by X is still given by (12) and (13) still holds. The 
expected log penalty of (12) is also the information of 9 given X. 

Once again an alternative decomposition to (13) employs the marginal distribution of 9 
predicted by the model. For the conditional probability p*(x.\uj) that X* = x given that the 
standard normal random variable 9* = uj, one has the decomposition 

P*(x,w) = (f>(u)p* (x.\u). 

Consider density prediction of 9 by (j). The actual density / of 9 satisfies 

/M = E(f* MX)) (19) 

for u) real, so that the expected log penalty is 

H*(9) = -E( log (j>(6)) = - j f(u) log (f>(u>)du! = [log(27r) + E(9 2 )}/ 2, (20) 

and the information of 9 is 

H(9) = - J f (w) log f(u)duj, 

so that H*(9) > H(9), with equality if the model holds. Once again, (15) holds, so that (16) and 
(17) follow. Essentially the same argument suffices to show that A(X|0) and A(0|X) are positive. 

For the normal case, there is some necessary relationship between the measure A(0|X) and 
the conditional variance <r(0|x) of 9 given X = x. A standard information inequality implies that 

2A(0|X) > E{9 2 ) - 1 - E{loga 2 {9' 2 \X)) 

(Rao, 1973, pp. 162-163). If the model holds, then E(9 2 ) is 1. 
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1.6 Mean Squared Error 

Given either the latent class or normal case, the latent variable 6 can also be evaluated in 
terms of mean squared error of prediction. One can compute the conditional expected value 
i?(0|X) of 9 given X to obtain the expected a posteriori (EAP) estimate of 8 (Bock & Aitkin, 
1981). The proportional reduction of mean squared error from prediction of 9 by i£(0|X) rather 
than by E{0) is then 

p\8\y,) = 1- E{o\8\y,))/a 2 {9). 


Computations can be simplified in practice by considering the weighted sum 

i 

W (x) = y^ajXj 
1=1 

for x in X. In all models, E(8 |x) is a function of W(x). Let 


i 

V{u) = JJ[1 + exp(aju; - 7 j)] _1 . 
1=1 


In the normal case, 


£(%) 


J 9ex.p[W(~x)uj]V(uj)(j)(u)duj 
f exp(W(x)cu)V(uj)(/)(cj)dcj 


Numerical computations can be performed efficiently via adaptive Gauss-Hermite quadrature 


(Haberman, 2006) with 9 integration points. In the latent-class case, 

E( 0 M = Ek=i T k exp[kL(x)r fc ] V (t a .)tt* 
ELi ex P [W{x)T k ]V(T k )ir* k 


Similar arguments apply to conditional variances. 


1.7 Estimation 

In practice, maximum-likelihood estimation may be used to treat the problem that the 
distribution of X is not known. Given use of maximum likelihood, estimates a* of a, and 7 $ of 7 * 
are obtained for the appropriate model. In the latent-class case, maximum-likelihood estimates 77 . 
of 7 Tfc are used to estimate 7 r^. In all cases, W(x) is approximated by 

1 

w (x) = 

i= 1 

and V(uj) is approximated by 

1 

= fj[l + exp(diUJ - q^)] -1 . 

1=1 
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The log likelihood is used for estimation of H*(X). Here the log likelihood function is 

n 

pr) = ^logr(X„) 

V=1 

for r in S. For the subset T of S corresponding to the model under study, the standard estimate 
of H*(X) is 

H(X) = 0 nk)~ l £{T ), 

where £(T) is the supremum of £(r) for r in T (Gilula & Haberman, 1994, 1995). 

In other calculations, p* is normally approximated by a maximum-likelihood estimate p in T 
such that £{p) = £(T), and p is approximated by p for p(x) equal to the fraction of v, 1 < v < n, 
such that X„ = x. One approximates Pi(u>) by 

Pi(u) = [1 + exp (-aw + Ti)]^ 1 , 


so that p*(x.\uj) is approximated by 

i 

1=1 

In the normal case, p(x) is then approximated by 


f*(bJ |x) is approximated by 


E(9 |x) is approximated by 


p(x) = j p(x.\u)(f>(uj)diij, 


f{u |x) = p{x\u)(/)(uj)/p{x), 


E{B |x) = 


f coexp[W{x.)u}]V(uj)(f>(uj)dijj 
f exp[W(~x)io]V(Lu)(/)(io)duj 


u 2 (0|x) is approximated by 


x JV - E(9\x)] 2 exp[W(x)u;]V{uj)(t)(uj)du 
a (U x = — 


f exp[W’(x)u;]V’(a;)(/)(a;)da; 

H*(X\9) is approximated by 

n r, 

H(x\9) = -(niy'Yllp'Ptv)}- 1 I P*0^-v |cu)[log p* (X„ \u>)\(/)(uj)duj, 


V=1 
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and H(9 |X) is approximated by 


n r. 

H(9 |X) = — n_1 5Z/ />|X,)[log/((j|X t ,)]dca. 


In the latent-class case, p(x) is approximated by 


f*{r k |x) is approximated by 


p(x) = 


f(Tk |x) =p(x|r fc )vrfc/p(x), 


l?(0|x) is approximated by 


cr 2 (0|x) is approximated by 


£,q | X N = Efc=i r fc exp[VF(x)r A; ]l/(r fc )% 
Efc=i exp[iy(x:)Tfc]l/(Tfc)7rfc 


cr w x = 


Y. k =il T k ~ E{d\x)} 2 exp[W{x)T k ]V{T k )7r k 
J2k=i ex.p[W(x.)T k ]V{r k )7r k 


H*(X\9) is approximated by 


£(X \ 0 ) = -(n/)- 1 ^[/(X„)]” 1 ^r(X t ,|r fc )[logr(X t ,|r fc )]7r fc , 


and H(9 |X) is approximated by 


H(0 |X) = -n-^/VfclX^flog/VfclX,,)]. 


In all cases, A(X|0) has estimate 


so that A(0|X) has estimate 


One then estimates H*(9 ) by 


The estimate of ^(^(^jX)) is 


A(X|0) = tf(X)-ff(X|0), 


A(0|X) = JA(X|0). 


H{9) = H(9\X) + A(0|X). 


E(a\9\X)) = n- 1 ^d 2 (0|X, u ), 
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the estimate of E(9 ) is 


the estimate of a 2 (9) is 


a\9) 


E(0) = n~ l Y J E(e |X V ), 

V=1 


E{<?\e |x))+ n~ i - m\\ 

V=1 


and the estimate of p 2 (9 |X) is 


p 2 (9\X) = l-E(a 2 (9\X))/d 2 (9). 


2 Empirical Results 

Results for the Praxis data described in the introduction are summarized in Table 1. Recall 
that there are I = 45 items and n = 8, 686 examinees. Computations employed Fortran 95 
programs designed by the author for use in item-response analysis. The column with label KR 
is the Kuder-Richardson estimate (Kuder & Richardson, 1937) of the reliability of LF(X) The 
latent-variable cases use K = 5 and = (k — 3)/2 1 / 2 , so that 9 would have mean 0 and variance 
1 if all latent-class probabilities 7were equal. 

To illustrate relationships between columns, it is helpful to observe that 

A(0|X) = 45[R(X) - R(X|6>)]. 

For instance, in the normal 2PL case, 0.917 is 45(0.592 — 0.571), except for rounding error. In 
the computation of p 2 (0|X), it is helpful to note that the estimated variance cr 2 (9) of 9 in the 


Table 1 

Results for OPL, 1PL, and 2PL Models 


Model 

R(X) 

H(X\9) 

A(0|X) 

p 2 (0|X) 

KR 

Latent OPL 

0.669 

0.652 

0.756 

0.793 

0.816 

Normal OPL 

0.669 

0.652 

0.773 

0.786 

0.816 

Latent 1PL 

0.596 

0.578 

0.828 

0.821 

0.816 

Normal 1PL 

0.596 

0.578 

0.849 

0.817 

0.816 

Latent 2PL 

0.591 

0.572 

0.888 

0.846 

0.829 

Normal 2PL 

0.592 

0.571 

0.917 

0.838 

0.830 
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normal 2PL case is 1.002, a value quite close to the ideal value of 1 obtained if the model is correct 
and sampling error is negligible. The estimated conditional variance a 2 {9 |X) is 0.162, so that 
p 2 (9 |X) = 1 — 0.162/1.002. The similarity between the columns for KR and /S 2 (0|X) is typical. 

As evident from the values of H (X) and H(X.\9), the 0PL model is considerably less successful 
than is the 1PL model, and the 1PL model is a bit less successful than is the 2PL model. For the 
same basic model, performance of the normal and latent-class cases are quite similar. The normal 
cases lead to a bit larger values of A(0|X) and a bit smaller values of p 2 (0|X). The variations in 
reliability are rather modest, although there is some advantage with use of the 2PL model. 

One method to assess the practical effects of results is to consider shortening the test and 
examining how the normal 2PL results compare to other reported results for 45 items. For 
example, with the first 30 items, p 2 (9 |X) is reduced to 0.773, a value a little lower than that 
reported for either 0PL model. The value of A(0|X) is now 0.746, again a value a bit less than for 
the 0PL cases. With the first 35 items, /> 2 (0|X) is 0.797, a value slightly larger than for either 0PL 
model and somewhat smaller than for either 1PL model, and A(0|X) is 0.804, a value somewhat 
larger than for either 0PL model and somewhat smaller than for either 1PL model. With the first 
40 items, p 2 {9\X) is 0.824, a value a bit larger than for either 1PL model, and A(0|X) is 0.877, a 
value somewhat larger than for either 1PL model. Thus the results do provide some suggestion 
that the gain from a 2PL rather than 1PL model is sufficient to have an appreciable effect on the 
length of a test required to achieve comparable precision in prediction of the latent parameter 9 
from the observed X. 


3 Conclusions 

These results are encouraging to the extent that they suggest a method of constructing an 
ability parameter that does not depend on model validity. The empirical results for the example 
considered suggest that the effects of model error are modest in terms of use of the ability 
parameter as a measure of examinee ability. On the other hand, the results also suggest that the 
old-fashioned total score gives results rather comparable in terms of reliability to results obtained 
by use of the theory of item responses. To the extent that these observations can be confirmed 
by analysis of a variety of test results, it may well be the case that the effects of model-based 
inference based on wrong models may often be sufficiently small not to cause significant problems 
in practice. This conclusion is most likely to hold for well-constructed and relatively long tests in 


14 



which Rasch or two-parameter logistic models are employed, for estimated item discrimination is 
likely to vary less, the effects of the latent ability distribution on the posterior ability distribution 
are reduced, and the model error is relatively small. One major aid is that the estimation of the 
ability parameter still depends on the total score or on a positively weighted sum. Because the 
model leads to a posterior distribution of the ability parameter given the responses that is a 
smooth function of the total score or weighted total, standard statistical results for functions of 
sums of independent variables indicate that it is indeed reasonable to anticipate that results based 
on estimation of the ability parameter are quite comparable to those for the total or weighted 
total. 

It should be noted that the approach used in this paper can be considered for multivariate 
ability distributions as well as for univariate ability distributions and it can be applied to 
polytomous items. No inherent change in methodology is required. 
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